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ABSTRACT 

The experiences of novice users (freshman students at 
Ohio University) searching OCLC*s FIBSTSEARCH set of databases were 
studied. The 13 subjects were asked to complete a search on the 
FIRSTSEARCH databases; and data were collected through observations, 
administration of a questionnaire, and a thinking-aloud protocol. 
Data were analyzed to describe a-^-d categorize the subjects^ 
experiences with the user interface. Results indicate that the novice 
user found the FIRSTSEARCH system user-friendly, with an average 
approval rating of seven out of nine on the Questionnaire for User 
Interface Satisfaction (QUIS) developed at the University of Maryland 
to measure human-computer interaction. Eight tables present study 
findings. Two appendixes contain the script for recruiting volunteers 
and a transcript of the search observations. (Contains 21 
references .) (SLD) 
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THE USER INTERFACE OF FIRSTSEARCH ON THE IBM: A QUALITATIVE STUDY 
ON NOVICE USERS 



This study examines the experience of novice users (freshmen 
Ohio University students) searching OCLC's FIRSTSEARCH set of 
databases. These subjects were asked to complete a search on the 
OCLC's FIRSTSEARCH set of databases, and data was collected via 
observation, questionnaire, and a "think-aloud protocol". Data was 
analyzed to describe and categorize the subjects' experience with 
the user interface. The results of the study indicated that the 
novice user found the FirstSearch system user-friendly with an 
average approval rating of 7.0 out of 9 on the QSIS questionnaire 
rating human-computer interaction. 
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FIRSTSEARCH 



FIRSTSSARCH is a set of Online databases offered by the Ohio 

Online Computer Library Center (OCLC) , to member libraries at a 

reduced price per search. The databases contained within 

FIRSTSEARCH are: 

OCLC Online Union Catalog, ERIC, GPO monthly catalog. 
Consumer Index, MiniGeo Ref, Biosis/FS, Readers Guide to Periodical 
Literature , Periodical Abstracts , Readers Guide Abstracts , 
Newspaper Abstracts, Humanities Index, Business Periodicals Index, 
Wilson Business Abstracts, PAIS Decade, Biography Index, 
PsychFIRST, socioAbs, BusinessOrgs , Disclosure Corporate Snapshots, 
Fact Search, Book Review Digest, Cumulative Book Index, Art Index, 
Essay and General Literature Index, Business Organizations, 
Agencies and Publications, Education Index, Index to Legal 
Periodicals, Library Literature, Social Science Index, Applied 
Science and Technology Index, Biological and Agricultural Index, 
Concise Engineering Index, and General Science Index/ 



The basic set-up is one in which libraries may purchase blocks 
of searches at reduced prices. The price of the searches is on a 
sliding scale with the cheapest rate being 45 cents per search for 
the largest number of searches purchased (80,000 searches) and the 
most expensive, 90 cents per search when only 500 searches are 
purchased. Searches can be performed from personal computers, 
public access computers or OCLC workstations and the set-up is 



^ O'Leary, Mick. Database Review, "FirstSearch Takes the 
Lead", Information Today , February, 1992. 
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designed to augment existing libraries offerings to the patron.^ 

There are numerous benefits to FIRSTSKARCH compared to just 
buying time for online database searching, such as — extensive, no 
cost "help" functions which facilitate successful searching by the 
novice or amateur, easy-to-use search features/directives, 
searching any record field, limiting by date, record type, etc., 
and index expansion. The computer : user interface is designed to be 
easy-to-use and easy-to-follow.'' Results from the searches are 
displayed in truncated form for browsing by the end-user, full 
record display is available for desired records chosen. 

The main problems which have been noted by both end users and 
the designers of the system are — the seemingly long wait for 
displays to be brought-up, the truncation of titles, and the one- 
by-one full record displays which require a great deal of end-user 
time to be reviewed or downloaded.* What has not been 

demonstrated yet is the "user-friendliness" of the computer :user 
interface in the working environment of OCLC-member libraries which 
have subscribed to FUtSTSBARCH because it appears to be a good 
investment and a beneficial service to its patrons. 

During an active search with a "user-friendly" computer 



^ Gupta, Usha and Lutishoor Salisbury. "Is FirstSearch Really 
Attractive", College and Research Libraries News , vol. 53, no. 7 p. 
461-368, July/August 1992. 

^ Campbell, Nancy, "FirstSearch Catalog Introduced", p. 19-28, 
OCLC Newsletter September /Octob er 1991. 

* Benefiel, Candace R. and Steven Smith, guest coluianists, 
"FirstSearch: A Survey of End-Users", p. 16-18, QCLC Micro , December 
1991. 
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interface, a client/end-user will often formulate and modify their 
search-strategy in xaid-search almost without being aware cf what is 
occurring due to the help the interface and search program 
generates as cues to the user. Their grasp of the needs to 
formulate and modify the search increases as they find information 
pertaining to their data requirements via positive feedback 
(relevant information) from appropriate search commands*. To 
mediate any lack of search success in electronic bibliographic 
access, search strategies and interface capabilities need to be 
modified to cover as many contingencies as possible in indexing 
structure, this modification can be learned through experience or 
through bibliographic instruction. There is always a need for 
improving the indexing methods for bibliographic access to better 
meet information retrieval needs.* There is also need for a 
continuous assessment of how the user perceives and how the user 
utilizes the information accessing product.^ How well the 
computer : searcher interface fits the needs of a novice searcher is 



^ Dalrymple, Prudence W. , "Retrieval by Reformulation in Two 
Library Catalogs: Toward a Cognitive Model of Searching Behavior". 
Journal of the Am erican Society for Information Science, 41 (4): 
272-281, 3.990. 

* Stoan, Stephen K. " Research and Information Retrieval Among 
Academic Researchers: Implications for Library Instruction", 
Library Trends , Vol. 39, no. 3, p. 238-257, Winter 1991. 

^ Efthimiadis, Efthimis N. and Stephen E. Robertson, " Feedback 

and Interaction in Information Ret riev a l" , p. 257-272 from 

Perspectives in Information Management , London, Butterworths 1989. 



not well understood.^ 

Now that FIRSTSBMRCSI has made possible direct public access to 
online databases at highly reduced charges, what is needed is an 
<^xamination of how FIRSTSEAACH is received and how it is being used 
b> the public. 

PBOBUBM STATSHENT 

There have been great advances in electronic bibliographic 
information retrieval in the last ten years — CD-ROMs, Online 
services, and now OCLC's FIRSTSEARCH set of databases. FXRSTSEARCH 
is an online service aimed primarily at CD-ROM searchers who desire 
more currency in their retrievals. 

User satisfaction with CD-ROM's is rated high, yet studies 
have shown that training and experience are required to achieve 
the full potential use of the product. Despite this, users feel 
that they need neither training nor instruction for proper CD-ROM 
use.* Most CD-ROMs are marketed for use by a novice or 
inexperienced searcher and come with an interface designed to 
address common problems in the hope that it will increase search 
success and provide useful information for future product design. 
The same can be said for a product like FIKSTSBARCH. 

FIRSTSSARCH needs to be critically tested for product 

* Logan, Elisabeth, "Cognitive Styles and Online Behavior of 
Novice Searchers", Information Processing and Management. Vol. 26, 
No. 4, pp. 503-510, 1990. 

^ Somporn, Puttipithakporn, "Interface Design and User 
Problems and Errors: A case Study of Novice Searcher:, EQ 30 
(Winter 1990): p. 196. 
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effectiveness^ practicality, and user satisfaction potential within 
the scope of other electronic information retrieval. It is up to 
the originator of an information access product to refine that 
product to meet user needs and should not require extensive 
computer experience to operate. There has to be critical analysis 
of the computer interface and how it relates to user satisfaction 
to keep the product viable in the information access market. 

LITSRATUSE RKVIEM 

Patricia Thomas has put together an excellent literature 
review on user interfaces and this section draws heavily on her 
work." Alan Kay has done much of the preliminary research on the 
computer interface: user relationship. He admonishes designers not 
to merely mimic the use of paper information access means to the 
software interface, but to formulate "user illusion"." He 
stresses that an interface should be so easy that even a computer- 
innocent child could understand it. Even "user-friendly" 
interfaces require that the users be identified and catered to, so 
that in actual use, some interfaces may be extremely user-friendly 
to an experienced user but hard to work effectively to a completely 

Zink, Steven, "Toward Mere Critical Reviewing and Analysis 
of CD-ROM User Software Interfaces, " cd-RQM Professional 4 (January 
1991): 16-22. 

Thomas , Patricia , "The User Interface of ERIC on the 
Macintosh: A Qualitative Study of Novice Users", Master's Research 
paper, Kent State University at Kent, Ohio, 1993. 

" Kay, Alan. "User Interface: A Personal View," chapt. in The 
Art of Human-Computer Interface Design , ed. Brenda Laurel, 191-207. 
Reading MA: Addison-Wesley 1990. 
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novice searcher." Perhaps "user-friendly" should be replaced by 
"intuitive" to describe software which is self-explanatory.^* 

Shneidexraan lists eight basic underlying principles of good 
interface designs 

1. Strive for consistency 

2^ Enable frequent users to Ufc>a shortcuts 

3. Offer informative feedback without confusing error messages 

4. Design dialog to yield closure giving users a sense of 
accomplishment and indicating the time to prepare their 
next actions 

5. Offer simple error handling, telling the user how to 
correct the error that occurred, not just that it occurred 

6. Permit easy reversal of actions 

7. Support internal locus of control: "make users the 
initiator of actions rather than the responders" 

8. Reduce short-term memory load by using the screen to 
display frequently needed information. 

There has been no quantifiable means by which to measure "user- 
friendliness" and good interface design with the end-user in mind. 
Shneiderman lists five measurable human factors for evaluation: 

1. Time needed to learn 

2. Speed of performance 

" Nicholls, Paul, and R. Van Den Elshout, "Survey of Databases 
Available on CD-ROM: Types, Availability and Content", Database , 13 
no. 1, (1990): p. 18-23. 

Helgerson, Linda W. " Some Techniques for Observing Users," 
In The Art of Human-Compu ter Interface Design ^ ed. Brenda Laurel, 
p. 85-90, Reading MA: AddAson-Wesley , 1990. 
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3. Rate of errors by users 

4. Subjective satisfaction 

5. Retention over time" 

This study concentrates on items three and four of this list. The 
remaining evaluation factors are outside the scope of this study. 
Further studies will help in the quantifying and refining of this 
list. 

Qualitative analysis (also identified as ethnography, 
naturalistic inquiry, case studies, fieldwork, field studies, and 
participant observation)" is helpful in studying phenomena in 
which the end-result is to describe what exactly is happening 
through the course of the interaction. These methods are 
especially applicable to evaluate library applications when the 
presence of a technology is meant to augment existing services 
rather than totally replace them. The purposes of qualitative 
studies is to describe and understand normally occurring events and 
not prediction, control, etc. as in quantitative experimentation. 

Triangulation will be employed to guard against subjectivity 
and researcher bias which are inherent in much qualitative work. 
In this study, triangulation employs three means of addressing the 
measurement of a single occurrence so that any bias is averaged 
out. The approaches employed here are — direct observation, think- 

Shneiderman, Ben. Designing the User Interf ace; Strategies 
for Effective Human-compute r Interaction. Reading, Mass.: Addison- 
Wesley, 1987. 

" Donald, Ary, Lucy Jacobs, Asgar Razevieh, Introduction to 
Research in Education , 4 th. ed. , (Fort Worth: Holt, Rinehart and 
Winston, Inc., 1990), p. 444. 



aloud protocol, and a questionnaire. All of these techniques are 
standard measures employed in qualitative analysis and have proven 
to be satisfactory.^^ The questionnaire employed will be "The 
Questionnaire for User Interface Satisfaction (QOIS)" which was 
developed by the Human-Computer Interface Laboratory at the 
University of Maryland specifically to measure user satisfaction 
with the interface of the computer software. 

BESEARCH QUESTION 

In qualitative research, there is no hypothesis but rather a 
research question. The research question here is how effective is 
the FISSTSEARCH user-computer interface. 

METHODOLOGY 

This study employed qualitative techniques to document the 
experience of novice end-users when searching FIRSTSBASCH at Ohio 
University. No claims of general izability are made. Case studies 
are used to assess the FIRSTSEARCH user interface. 

Samp le; This study used volunteer subjects from the freshman 
class at Ohio University. This is a purposive, non-probability 
sample where the volunteer group is judged to be typical of novice 
users . 

Instrumentation ; Triangulation of data collection was used to 

" Connell, T., 1991, "Library Subject Searching in an Online 
Library Catalog: An Exploratory Study of Knowledge Use", (Doctoral 
Dissertation, Univ. of 111. at Urbana-Champaign) . 

Shneiderman, Ben. 
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reduce researcher bias. The three methods employed were — 
systematic observation, think-aloud protocol, and a questionnaire. 
The search question used for all thirteen samples was the same. 

The think-aloud protocol is a method of data collection 
whereby subjects are asked to engage in their normal pattern of 
activity, but to verbalize their thoughts as they occur. The only 
response of the researcher is to encourage the s\ibject to think 
aloud should they lapse into silence.^' 

Procedures and Design ; This qualitative study used a small 
group of student volunteers from the freshman class from Ohio 
University who were given a search question to be used on 
FZSSTSEASCO. Each individual session lasted about one hour and 
participants were informed that time was not an issue. A script 
(see Appendix A) will be used to decrease variation in 
administration of session. The subjects were informed that their 
session would remain confidential and that software was being 
judged, not their performance. They were advised that they could 
stop the session at any time should they feel uncomfortable. 
Materials on hand were the FIKSTSEARCH catalogue and notebook and 
the FISSTSEARCH user guide. Questions were solicited before and 
after, but not during search. 

The researcher was seated in view of the volunteer. The 
volunteer's use of sources, and non-verbal behaviors were noted. 
At the end of the search session the QUIS questionnaire was 



Patton, Michael Quinn, " Qualitative Evaluation Methods'^ 
Beverly Hills, Sage Publications, 1980. 
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administered. 

Sign-up sheets were placed both at the Reference Desk and in 
the Student Meeting Room in Alden Library at Ohio University in 
Athens, Ohio. There were no students signed-up on the sheet 
located in the Student Meeting Room, but there were fourteen names 
listed on the sign-up sheet located at the Reference Desk. 
Attempts at contacting all fourteen were made but only thirteen 
agreed to participate in the study. Students were asked when in 
the coming week they would be available to spend about an hour in 
the library using FirstSearch. 

The appointments were made and kept, except for two instances 
when the students forgot they had prior commitments and had to 
reschedule their search time. The appointments were all in the 
afternoon, usually two or three/day. The students were all on 
time. They were greeted and given oral and written directions 
concerning basic computer use (i.e. "Return" signals the computer 
to carry-out an action/command, direction arrows for moving around 
the screen, etc.) and the general directions included in the 
consent packet. Students were asked if they had any questions 
after each phase of the administration of the preceding sheets or 
on the upcoming computer searching. Questions were answered and 
then the search question was given and explained as to what was 
desired in the search process and results. The search question was 
as follows: "I am concerned about obesity in young people. What is 
the current thinking on teaching young children (preschool and 
kindergarten through third grade) about nutrition and exercise?" 

12 



once the Search for this information was begun questions were 
discouraged and not answered. After the search, the volunteers 
were asked if they had any questions, if there were any the 
questions were answered, then the questionnaire was given. After 
the questionnaire was filled-out the novice searcher was asked 
informally how they perceived the FIRSTSEMRCH system and the search 
appointment was terminated. 

Data from the visual observation and "talk-aloud" tapes were 
integrated to give both oral and visual explanation as to what was 
occurring as the search progressed (see Appendix 2). General 
problems in the search were enumerated and tabulated. The 
questionnaire was scored and a group count and sub-group tallies 
were arrived at. 

Data analysis : included description of users, their background 
and their past experience with searching. The think-aloud 
protocols were compared with each other and with the data gleaned. 

SIGMIPIOUICE 

This study aids in the assessment of the interface used in 
the FIRSTSEMRCS set of database. This may help OCLC refine and 
further develop their product. 

RESULTS 

Of a total of thirteen searchers 31% had problems with typing 
in the password correctly with no screen feedback as to correctness 
of typing, 8% had problems with typing in the authorization code. 

In the search process 54% had problems in typing in the search 

13 
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phrase correctly, but these error (s) could be corrected by reading 
the screen and then backspacing to make corrections. However 31% 
encountered a 'glitch' or problem in the system in which the system 
would not backspace and the searcher would have to log-out of the 
system and restart-up on a totally new search. Printing caused the 
most confusion and took the most time. There were numerous keying 
errors but those could be remedied. All volunteer searchers 
completed this process. 

Table 1.1: PSOBLSIS 
LOGGING-IN COUNTS 
typing: password 4 
author i z at i on 



topic area choice 
SEARCH 

Correctable 

Non-correctable 

Punctuation 
PRINTING 

Failure to achieve 

Miss-cued* 
*(i.e. wrong keys punched) 



1 
4 

7 
3 
1 

0 

10 



PERCENTAGES 
31% 
8% 
31% 



54% 
23% 
8% 

0% 
77% 



ANALYSIS OP RESULTS 
LOGGIHG OH-" The main problems which occurred in the logging- 
on process were mistakes in typing or the keying in of characters. 
In novice users who have limited experience on a typewriter or 
keyboard, this is a logical mistake. Of course the FIRSTSEARCH 
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protocol having the passwords and personal authorization codes 
masked on-screen may improve secrecy, but does not help a novice 
typist or keyboard-user read their screen for typos and missed 
keys. Since PIRSTSEARCH is used on a personal computer station 
where semi-privacy exists, the need for protocols and keying 
measures insuring such high degrees of confidentiality, personal 
control, and secrecy may not be a high priority if it causes 
problems in time usage and frustration in users logging-on to the 
system. 

TOPIC AREA-— The choice of topic area seems to be a problem of 
users who were not acquainted with the search question concerning 
obesity and youth. Rather than attempting to further modify my 
subject population by requesting only for freshman majoring in 
anthropology, health sciences, or sociology (the subject/major area 
within in which the search question would probably fall), the 
experimenter chose to maintain a subject group of novice users of 
undetermined background and knowledge area. Matching of search 
question topic area and subject background in determining searching 
effectiveness was beyond the scope of this study. Volunteers for 
this research project were all given the same question with which 
they had had no previous acquaintance. Because of this lack of 
knowledge concerning the search question they basically had no 
framework on which to make a choice in subject area or database 
selection. This facility of search refinement is enhanced by use 
and familiarity with the system, familiarity with the desired 
subject matter, and/or more acquaintance with a list of the 

15 
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database descriptions. Novice users just have an innate 
disadvantage in this area. 

SEARCHIl^*- Within the search process itself, the main 
problems encountered were difficulties arising from the typing or 
keying-in of characters from the keyboard. Perhaps the use of a 
mouse and "windows" for command choices VTOuld eliminate some of 
these keying problems. No Macintosh computers were available at 
Ohio University's Alden library to test this idea. This is another 
area where the novice searcher/computer user has an innate 
disadvantage over the typist or well-versed computer user. The 
main problem arose when the system would not allow backspacing to 
correct perceived typing errors. Since this lack of correctability 
arose only 43% of the time when typing or keying mistakes occurred, 
it seems to be a 'glitch' in the system. This problem will have to 
be handled wherever the difficulty originates, either locally or at 
the riRSTSBARCH site.^"" 

ANALYSIS OF QillS QOESTIONIIAIRE 

The PUIS questionnaire was administered to all thirteen 
volunteer searchers, and all thirteen completed the questionnaire. 
This researcher used past experience on computers as an automatic 
categorizing method. Amount of computer experience was used to 
form groups because the degree of facility of use may influence how 
a user responds on the QUIS questionnaire. Although most of the 



'''' Snure, Karen, "The FirstSearch Experience at the Ohio State 
University", Library Hi Tech ^ issue 36-9:4, p. 31 , 1992. 
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thirteen had had some computer use^ none had any experience with 
online or CD-ROM searching. The following table lists the total 
numbers and percentages per group and for the entire volunteer 
subject group. 

One volunteer (Group 1) had a total of experience on 7 
different forms of computer software^ hardware, and other devices 
in their computer experience. The next subgroup (Group 2) had 
experience with a total of 13 different computer software, hardware 
and other devices. Another group (Group 3-4) consisted of four 
volunteers who had had 3-4 types of computer systems in their 
experience repetoir. They had diverse experiences of different 
devices on these systems for a total of 17 forms of computer 
software, hardware and other devices in their experience. There 
was one volunteer novice searcher (Group 5-6) who claimed to have 
experience on 5-6 computing systems with an acquaintance of a total 
of 10 computer software, hardware devices, etc. 
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Table 1.2: PAST EXPERIENCE 



GROUPS 



PAST 




1 




2 




3-4 




5-6 




EXPERIENCE 


M 


% 


M 


% 


M 


% 


N 


% 


% 


Keyboard 


1 


7.7 


7 


53.8 


4 


30.8 


1 


7.7 


100 


Numeric Key pad 






2 


15.4 


3 


23 






38.5 


Mouse 


1 


7.7 


7 


53.8 


4 


30.8 


1 


7.7 


100 


Light Pen 




















Touch Screen 






1 


7.7 


1 


7.7 






15.4 


Track Ball 










1 


7.7 


1 


7.7 


15.4 


Joy Stick 






4 


30.8 


2 


15.4 


1 


7.7 


53.8 


Text Editor 




















Word Processor 


1 


7.7 


4 


30.8 


4 


30.8 


1 


7.7 


76.9 


File Manager 










1 


7.7 






7.7 


Electronic Spreadsheet 




1 


7.7 


1 


7.7 


1 


7.7 


23.1 


Electronic Mail 










2 


15.4 






15.4 


Graphics Software 


1 


7.7 






2 


15.4 






15.4 


Computer Games 


1 


7.7 


4 


30.8 


3 


23.1 


1 


7.7 


76.9 


Color Monitor 


1 


7.7 


4 


30.8 


2 


15.4 


1 


7.7 


61.5 


Time-share System 




















Workstation 




















Personal Computer 






5 


38.5 


4 


30.8 


1 


7.7 


69.2 


Floppy Drive 


1 


7.7 


7 


53.8 


4 


30.8 






92.3 


Hard Drive 






4 


30.8 


4 


30.8 


1 


7.7 


23.1 


Compact Disc Drive 






1 


7.7 


2 


15.4 






15.4 



Taken as a group these volunteers are assumed to be 
representative of any and all novice FIRSTSEMtCH searchers. 

The ratings on the questions of the QUIS questionnaire 
can be assumed to be Indicative of responses of novice searchers 
encountering the FirstSearc^h set of databases system. 
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Question 



Table 1.3: OVERALL OSER REACTIONS 

Rating 



overall Reaction to systea 

terrible. . .wonderful 



frustrating. . .satisfying 



dull. . .stimulating 



difficult. . .easy 



1 
1 

1 

1 



1 
2 



Inadequate power .. .Adequate power 



rigid. . .flexible 



2 

2 



2 
1 



1 
1 



3 

1 
1 
5 



1 
1 



1 
1 
1 
3 



1 
1 



1 

1 
2 



2 
2 



2 
2 



3 
1 



1 
2 



1 

3 



8 
1 



3 
1 

4 

1 



1 
1 



1 
2 
1 
4 



2 
1 



9 
2 



1 
2 



1 
4 



1 

2 



N.A. 



1 

1 



1 

1 



Group 



1 
2 

3-4 
5-6 
Total=13 

1 
2 

3-4 
5-6 
Total=13 

1 
2 

3-4 

5-6 

Total=13 

1 
2 

3-4 

5-6 
Total=13 

1 
2 

3-4 
5-6 
Total=l3 

1 

2 

3-4 
5-6 
Tatal=13 



The ratings of Group I's overall User Reaction was generally 
positive with an average of 8.3. The one '1' rating was concerning 
the flexibility of the system, and since FirstSaarch is strictly a 
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bibliographic citation retrieval tool, its flexibility is 
inherently narrower that other systems with broader, more numerous 
utilities. The ratings of the searchers (Group 2) with experience 
on two computer systems on the Overall User Reaction to the System 
was mixed #/ith an average of 7.3, and the range in scores being 
from 5... 9* So the ratings indicate that these searchers were 
generally favorable to the FirstSearch system. The broader spread 
in ratings and higher average in scores given in comparison with 
Group I's may indicate a greater selectivity, individuality, and 
sensitivity that >re experienced computer users have when 
assessing a new computer system. In Group 2, the ratings in 
reaction to the system had an average of 7.3. In Group 3-4, the 
section measuring the searcher's ratings to the Overall Reaction to 
the System showea a fair irange from 4... 8, with two "Not 
Applicable" for an average of 6.1. The wider range in scoring, 
albeit lower average may indicate a more selective and 
discriminating clientele. The two "Not Applicable" scores may 
indicate a doubt about that question being applicable to this 
computer system. For the group with experience on 5-6 computer 
systems (Group 5-6), their scoring on the Overall Reaction to the 
System was in the medium range consisting of scores of 4... 8, with 
an average of 6.2. The broader range and lower average in ratings 
correlate positively with the greater degree of computer system 
experience, leading to the conclucion that greater experience on 
computer systems leads to greater discrimination among computer 
systems offerings. The average of all the groups for Overall User 
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Reactions was 6.9. 



Table 1.4: OVERALL USSR RBACTION TO SCREBM 

Question Rating 

3 4 5 6 



1 2 

Charact«rs 

Hard to read . . . easy to read 



Highlighting 

not at all... very helpful 



Layouts 

Not at all... very helpful 



Sttqaeno« 

Confusing. . .clear 



1 
1 



2 
1 



8 



1 
2 



1 
1 
1 



1 
4 
2 
1 
8 

1 
2 
2 
1 
6 



Group 



N.A. 



1 

6 
2 
1 

10 



6 
3 
1 

10 



1 
1 



1 

2 

3-4 
5-6 
Total=13 

1 
2 

3-4 
5-6 
Total=13 

1 
2 

1 3-4 
5-6 

1 Total=13 

1 
2 

3-4 

5-6 
Total=13 



Group I's reaction to the Screen consisted of 8's and one 9 
for an average of 8.7. This very highly favorable reaction can be 
attributed to the fact that this group had very little experience 
for comparison of computer systems. Group 2's overall Reaction to 
the Screen had an average of 7.6. Group 3-4 's average for this 
section of questions was 8.2. This group had one "not applicable" 
response, the only such response from all of the subjects in this 
section of questions. This response can be attributed to the fact 
that subjects in this group (Group 3-4) had little basis and 
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background for comparison and assessment of layouts on the screen. 
FirstSMTCh is highly specific and specialized for both service and 
user groups, so uncertainty in responding to such a question is 
reasonable and bonifide. Group 5-6 's average was 8.5. The 
generally lower average scores of the groups correlates positively 
with greater experience on more computer systems. The average for 
of all of the groups was 7.9. 
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Table 1.5: 

OVERALL USER RSACTION TO TESNZMOLOGY HMD SYSTEM mFCffilCATIOIi- 



Questlon 

3 

Use of toniinology 
thrdagbout systea 

Inconsistent:. . .consistent 



Rating 



1 
1 
1 
3 



7 
1 
1 



2 
1 



2 
2 



N.A. 



Group 

1 

2 

3-4 
5-6 
Total=13 



Does the teralnology relate well to i 1 

tbe liork you are doing i i 122 2 

Unrelated. . .well related 11 2 3-4 

1 5-6 

1111212 4 Total=13 



Messages wttich appear on 
the screen 

Inconsistent. . .consistent 



1 

1 



1 
1 



1 
2 



3 
3 



1 
2 

3-4 
5-6 
Total=13 



Messages irtiich appear on 
the screen 

confusing. . .clear 



1 
1 



1 
1 
2 



4 
2 



1 
2 

3-4 
5-6 
Total=l3 



Does the c<»puter keep you 
inforaed about itbat it is 
doing 

Never. . .always 



1 
1 



1 
1 



1 
2 
2 



2 
1 



1 
2 

3-4 
5-6 
Total=l3 



ErroL aessages 

Unhelpful. . .helpful 



1 
1 



2 
1 
1 
4 



1 
1 
1 



1 
1 



1 
2 

3-4 
5-6 
Total=13 



In the Terminology and System Information section for rating 
the computer system, there was a wider range of rating measures for 
the entire Group 1 volunteers, — 7, 8, 9, for an average of 7.7, 



indicating a generally favorable reaction to the utility of the 
screen. The group with experience on 2 computer systems rated the 
terminology and System Information was rated beneficially — the 
range consisting of scores from 3... 9, for an average of 7.2. The 
lower scores (3 & 4's) were concerned with the fit of the 
terminology used to the work being done, indicating that when a 
computer user, no matter how experienced, is set to perform 
computer work with which they have little or no intellectual 
investment, they tend to be a bit more confused and error-prone 
than when they totally understand the subject matter of the task 
they are performing. The Terminology and System Information 
section for Group 3-4 consisted of a range from 5... 9, with an 
average of 7.5, with three "Not Applicable". The "Not Applicable" 
scores may indicate a perception that the novice users did not 
fully grasp the potential or even the benefit of such a system for 
searching-out bibliographic citations. The rating of Group 5-6 in 
the Terminology and System Infoinoaation fell in the medium range 
consisting of a range from 3... 8, with an average of 5.2. Since 
they purported to be more computer experienced they may have rated 
the system more rigorously. The average for all of the subjects 
was 6.9. 
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Question 



Table 1.6: OVERALL USSR REACTIOM TO LEARMIliG 

Rating 



Group 



12 3 


4 


5 


6 


7 


8 


9 


N.A. 


Itearning to operate 










1 




1 




1 




1 


1 


z 






Difficult. . .easy 1 








2 


1 




3-4 








1 








5-6 


1 


1 




2 


3 


4 


2 


Total=13 


Bxpioratlcm of features 








1 






1 


cy uTiox aiKi ezxx>r 




J. 


J. 


i. 


X 




X Z 


Discouraging. . . 


1 




1 


2 






3-4 


encouraging 1 














5-6 


1 


1 


1 


2 


4 


1 


2 


1 Total=13 


Romeaboring niiaes and 










1 




1 




J. 


J. 




X 


X 




O 
Z 


Difficult. . .easy 






1 


1 


1 


1 


3-4 




1 












5-6 


1 


2 


1 


3 


2 


3 


1 


Total=13 


Can tasJcs be performed 










1 




1 


xn a Swraxgnu roxivazu 




1 


1 


ii 
4 




1 


z 


wuftner 








1 


1 


2 


3-4 


never. . .always 






1 








5-6 






1 


2 


5 


2 


3 


Total=13 


Help iwseages on screen 










1 




1 


Confusing. . .clear 






1 


2 


1 


3 


2 








1 


1 




1 


1 3-4 










1 






5-6 








2 


4 


2 


4 


1 Total=13 


Supplmental reference Material 












1 


1 


Confusing. . .clear 




2 




2 




2 


1 2 








1 




1 




2 3-4 






1 










5-6 






3 


1 


2 


1 


3 


3 Total=13 



ERIC 



Group 1 rated the learning aspect of FirstSearcdi vei^y high, 
ranging from 7... 9, with an average of 8.0, indicating that the 
FirstSearch system is user friendly and helps novice users in the 
first-time search process. The seven was related to trial and 
error learning and since only one search was experienced by the 
searcher, trial and error learning was limited, so that a low score 
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was relevant. In Group 2, the ratings are more spread-out, but are 
still highly positive to the system's aid-giving capabilities. The 
score's range being from 3... 9, with four "Not Applicable", giving 
an average of 7.3. The wider range in scores indicating that 
learning even on a "user-friendly" system is still highly 
individualistic and no computer system can encompass the entire 
gamut of learning strategies necessary to fit all learning styles 
to a 'T'. In group 3-4 the range is wider, varying from 2... 9, 
with three "Not Applicables" for an average of 7.5. The greater 
number of "Not Applicables" indicating that those questions were 
not relevant to the searchers and hence to the purpose of this 
investigation. The questions dealt with "help messages" on the 
screen and Supplemental Reference Material. "Help messages" were 
not being tested-for and "Supplemental Reference Material (s)" were 
not b3ing solicited, so that the "N. A."s were totally well placed. 
For Group 5-6, the ratings were more clustered being from 3... 7, 
with an average of 5.2, indicating that in fairly experienced 
computer users, the aspect of learning from the system is more apt 
to occur with experience than in users who are relatively 
inexperienced and requiring more help from the computer system 
itself. The average for all of the subjects was 7.0. 

The ease of learning the FirstSearch system is critical, for 
expeditious searching on the system. Since much of the ability to 
construct a successful search is gained from past experience, it is 
necessary for the user to learn from the searching they have done 
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and to learn as they do it.' 



Table 1.7: OVERALL USSR REACTION TO SYSTBI CAPABILITIES 



Question 



Rating 



Group 



12 3 


4 


5 


6 


7 


8 


9 N.A. 




Systea Speed 










1 




1 


Too slow... fast enough 






1 


1 




5 


2 












4 




3-4 














1 


5-6 








J. 


J. 


C 

O 


O 


TOtai— ± J 


How reliable is tbe syBtem 








1 






1 


Very unreliable. . .very reliable 






1 


2 


1 


3 


2 










1 


1 


1 1 


3-4 












1 




5-6 








1 


4 


3 


4 1 


Total=13 


Syntem tend to be 












1 


1 


Noisy. . .quiet 




1 






1 


5 


2 












1 


3 


3-4 














1 


5-6 






1 






2 


10 


Total=13 


Correcting your mistakes 












1 


1 


Dif f icult. . .easy l 




1 




1 


1 


1 2 


2 




1 






1 


1 


1 


3-4 




1 












5-6 


1 


2 


1 




2 


2 


3 2 


Total=13 



Are the needs of botrh i 1 

experienced and Inmcperienced i i 3 i i 2 

users taken into consideration 21 1 3-4 

Never .Always 1 5-6 

1 13 4 2 2 Total=13 



The System Capabilities measurements for Group 1 was high, but 
had a narrow range consisting of from 7... 9, with an average rating 
of 8.2. The System Capabilities for group 2 were also rated high 
with a much broader range indicating a more varied response in 
perception of the system, the range encompassing sores of 2... 9, 

Shaw, Debora, "Nine Sources of Problems for Novice Online 
Searchers", Online Review ^ Vol. 10, no. 5, pp. 295-303, 1986. 
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and two "N.A."s^ for an average of 8.6. Again, since FirstSearcdi 

is a bibliographic retrieval system its capabilities are inherently 
less diverse than many other computer systems, this is a very high 
score for such a specific computer system. Group 3-4 contained a 
range of scores from 4 — 9, with an average of 7.7, with three "Not 
Applicable". The "N.A."s may indicate again that the questions 
about "help messages" and "supplemental reference material" were 
not germane to the search they performed, which they were not. 

Group 5-6 rated the System Capabilities mixed, the range of 3 9 

with an average of 6.6, indicating the varied perception of the 
systems capabilities. The average score was positively correlated 
with the degree of experience claimed by the novice searcher. 
Experience seems to predispose searchers to make more critical 
judgements of the computer system they are using. The overall 
average for all of the groups was 7.9 for this section of 
questions . 

NRITTEN COHNENTS 

The searcher in Group 1 had written comments concerning his 
experience with the FirstSearch set of databases, he thought that 
the Fir8l:Search system "was very beneficial to learn" and "easy to 
use" and that he might very well "use it again if he would need 
references for a paper" . The searchers in Group 2 had written 
comments that were indicative of how they perceived FirstSearch. 
Most (86%) indicated that they found the search easy to perform and 
that the system facilitates learning about it while operating it. 
One person complained about not being able to backspace to correct 
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typing errors. In the Written Coitanents section 75% of Group 3-4 

complained about the intimidating aspect of logging'-on with the 

masked Passwords and Authorization keys, but they were favorable in 

their overall assessment of FirstSearcli system of Databases, There 

were no written comments from the searcher in Group 5's encounter 

with the system. 

\ TABLE 1.8: AVERAGES OF GROUPS 

\ Groups 1 2 3-^4 5-6 X 

Overall Reaction 8.3 7.3 6.1 6.2 6.9 

to Sy Btes 

Reaction to Screen 8.7 l.S 8.2 8.5 7.9 

ReactioVi to 7.7 7.2 7.5 5.2 6.9 
Technolck'gy and 
System X^oraation 

Reaction Vo 8.0 7.3 7.5 5.2 7.0 
Learning 

Reaction to 8.2 8*6 7.7 6.6 7.9 
SystMi Capabilities 

Although there are no significant statistical findings for 
this table, there are qualitative analyses which are relevant to 
this research. Generally speaking, the more experience on computer 
systems a group had, the more discriminating were their analysis of 
their use of the FirstSearch product. The most stringent of these 
analyses was the section of questions concerned with learning. 
Learning, especially in regards to novice users is an extremely 
important factor for a computer product provider. The programming 
must be computer-friendly (read conducive to learning) so that 
novice users are successful in their initial use and will continue 
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using the product. The novice users in this research were all 
successful in their use of the FirstSearch Program and thought 
positively of it. 

SDMNftRY/OOilCLDSIOIIS 

Froia the data gleaned during this investigation, it seeias that the 
FirstSearch systems has a relatively user-friendly computer 
interface. 

Perhaps the most telling question on the questionnaire 
was the last question in reference to how well the system satisfied 
the needs of both an experienced and inexperienced user. 

Group 1 , the most computer system ignorant searcher rated 
FirstSearch an 8. A positive assessment of the "user-friendliness' 
of the system. Group 2, consisting of seven users with experience 
on 2 computer system rated the FirstSearch system from 5... 9, with 
an average of 7.0. Signifying that as a group they rated it 
positively on user-friendliness. Group 3-4 rated this 

questionnaire with 5... 9 for a group average of 7.7. Group 5-6 
ratings was one 3 . Indicating perhaps , that more experience is 
correlated with more discrimination when dealing with computer 
systems. Comparison of the groups with each other yields little 
worthwhile data, except that as a whole the entire group on average 
ratings were 6.8, which is a positive indication of searcher 
appreciation. 

The total of all averages from all of the questions and all of 
the sections from the QDI3 questionnaire for all of the groups was 
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7.0, indicating a moderately strong approval rating for the 
PirstSaarch system. 

There are frequently a host of variables which can affect the 
processing and results of an investigation, especially one 
involving human subjects. These extraneous variables are often 
unknown at the time of the research. Usually the effects of these 
extraneous variables are negative in nature." Some of the 
examples of the extraneous variables and cause for further research 
in this analysis of the FirstSearch set of databases may be: 

1) . the basic problem of a novice searcher searching for subject 
material with which they have had no previous contact and have no 
conceptualization of what they are needing/ looking for/ etc. could 
result in disinterest, confusion, and/or increased inability to 
follow the directions in the computer program. 

2) . The problem of stress, typing skills, etc. affecting how an 
individual functions in an experimental environment by increasing 
nervousness, frustration, inability to follow logical directions, 
etc., resulting in a poor performance on an otherwise easy computer 
exercise. 

3) . the functioning of a relatively new computer system where all 
of the functions of the system may not yet be simplistic enough for 
novice searchers. A novice searcher who finds a computer program 
confusing or intimidating may not pursue further work (and learning 
and thereby refining of computer skills) on that same program. 



It is conceivable that any one or all of these extraneous 
variables could have been operating in this investigation, and 
possibly more as yet undiscovered effects of other unknown factors. 
Despite these potentially disabling effects, PiratSaarch still 
scored very high in "user-friendliness". In conclusion, based on 



" Donald, Ary, Lucy Jacobs, Asgar Razevieh, " Introduction to 
Research in Education . 4th. ed.. Fort Worth: Holt, Rinehart, and 
Winston Inc., 1990, p. 311-317. 
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this small amount of qualitative data and despite the inherent 
biases in the qualitative research, the data indicates that the 
PirstSaarch set of Databases as a system constitutes a novice-user- 
friendly system for bibliographic citation searching. 



RBOOMNEMDATIOliS 

Heckel noted that the best designed user interface will be the 
product of a designer who has a clear picture of the most likely 
end-use" and the foremost concern of a software designer should 
be clear communication,^* Since the end-user in this investigation 
was a novice searcher, clear, concise and simple directions are of 
the utmost importance. In addition, the research process and the 
results reviewed suggest several options for a network such as OCLC 
to better serve their clientele. 

1) Make sure the system runs as smoothly as possible, all keys and 
their functions are delineated clearly, no obscure functions 
occurring because of miscued keys. A novice searcher is often a 
young and inept keyboard user, it is important that they are not 
penalized for a lack of finger dexterity. 

2) In offering subject area decisions, give examples of databases 
within that area rather than requiring them to make educated or 
likely guesses. It will only confuse them and make them more 
unsure of their own judgment in the search process. 

2) Expedite the printing and downloading capabilities within your 
system, perusal should be at the user's discretion and not take-up 
valuable user time and library facilities. 



" Heckel, Paul, The Elements of Friendly Software Design . New 
York: Warner Books, 1984. 

^* Zink, Steven D., "Toward More Critical Reviewing and 
Analyses of CD-ROM User Software Interfaces", CD-ROM Professional , 
January 1991, p. 18. 
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4) Don't try to sell your wares where they are not necessarily 
needed. Most of the FlrstSeurcll databases are available on CD-ROM 
at little or no cost to the searcher, the currency of FirstSaarch 
is the commodity which should be emphasized and be the selling 
point. There is little point in creating a group of users which 
are antagonistic to your system because they used a system and 
spent their money on goods which did not fulfill their needs, as 
Pu ttpi thakpor n (1990) obser ved""* . 



Puttpithakporn, Somporn, "Interface Design and User Problems 
and Errors: A Case Study of Novice Searchers", RQ, Winter 1990, p. 
195. 



SCRIPT rOR RBORUITED VOLOMTEBRS Al^pmidix A 



I want to do research on FIRSTSBMtCH on the IBM computer. I 
waat to do this because user studies are important for the smooth 
implementation of computer systems in libraries and information 
centers, I would like you to take part in this project. If you 
decide to do this, you will be asked to perform a search of the 
FIKSTSBARCH set of databases on the IBM computer here at Alden 
Library. As you conduct your search you will be observed. You 
will also be asked to think aloud, and you will be tape recorded. 
After the search, you will be asked to complete a questionnaire. 
This entire process will take about an hour. 

The risks associated with this project are minimal. The 
experience should not be unpleasant unless you are very frightened 
of computers. As with the use of any computer, exposure to 
hazardous rays is possible. 

Your name and phone number will be used only to set up the 
search time. All information will be held in confidence. Audio 
tapes will be erased after transcription. You may listen to the 
audio tapes if you wish. 

If you take part in this project you will gain experience with 
IBM applications, with FIKSTSEMRCH searching, and with performing 
a reference search. Taking part in this project is entirely up to 
you, no one will hold it against you if you decide not to do it. If 
you take part, you may withdraw at any time without penalty of any 
kind. There will be a stipend of $10.00 if you, the volunteer, 
complete the search and questionnaire. 

If you want to know more about this research project, please 
call me at 593-5724 or you may contact Dr. Carl Franklin at 1-292-- 
7746. This project has bean approved by Kent State University 
rule's for research, please call Dr. Eugene Wenninger, telephone 
(216)-672-2070. 

You will get a copy of this consent form. 



Sincerely, 



C. Crysteen Cooper 
Graduate Student 

School of Library and Information Science 
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1/22/93 

12:10 
12:11 

12:12 

12:13 
12:14 
12:15 



1/22/93 

2:10 
2:11 



2:12 



2:14 



1/23/93 

1:30 

1:32 
1:34 

1:35 

1/24/93 

1:55 
1:56 

1:57 

2:05 



TRMISCRIPTIOIfS OF SBARCHBS Appendix B 

JUSTIN 

On Search 

"let's see, Fse&rch is the next 
direction" . 

"Oh damn, lets get this blasted password 
right". 

"BRIC is a social sciences database" . 
"Mitunro. . .Obesity and youth*'. 
Some confusion over hitting ''return" after 
every command . Genera 1 ly read /stud i ed 
screen for a minute or so before 
committing any action. 

GLENN 

On Search 

"Is ERIC a humanities database, evidently 
not ( after reading 1 ist of databases 
included in Humanities area databases),, 
lets try Social Sciences..." 
Mis-searched "su: obesi ty and vouna 
peo p3,g "-no hits. Researched under correct 
nomenclature/search terminology . 
accompanied by various curse words. 
Mis-punctuated "limiting of 1990-1993 as 
1990. . .1993." 

So had to re-search. Again generally 
studied/read screen for a minute or so 
before taking any action. 

JESSICA 

Logged-on easily, little or no 
problems/questions . 

"Lets see Social Sciences Area was where 
she (experimenter) told me to find ERIC". 
"Damn. . . su: Obesi ty and Vo^t.f^ and for 
second search.. , L,1 and 1990-1993 , that 
should be right." 
Search Completed. 

CHRIS 

Logged on quickly and easily. 
"Mmmm. . . lets try humanities for area, no, 
I'll try Social Sciences, yes, that's 
right." Chose wrong topic area but self- 
corrected . 

Mistyped Youth and Obesity , system would 

not backspace to allow for corrections, so 

had to relog-on and perform 

another search. 

Search completed correctly. 
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1/24/93 

3:10 



3:12 



3:15 



CINDY 

"damn, password didn't work, oops, I used 
the wrong password rather than the 
authorization number" , searcher was 
confused by the long wait for OCLC to 
correct prompt for a new authorization 
cue. 

Mis-searched due to incorrect spelling in 
the search phrase, system would not 
backspace to allow for corrections, so had 
to perform two searches. 
Finished search 



1/24/S3 

4:10 



4:11 
4:15 



KIBSTEM 

logged-on correctly, although mistyped-in 
password, but system allowed backspacing 
to make corrections. 

"lets see, fsimrch, then my own password 

and authorization codes". 

Search completed with few or no 

hesitations. 



1/25/93 

12:30 
12:34 

12:35 



MICHELLE 

Logged-on o.k. 

"Mmm, how to get only years wanted, yes, 
use L,l and then 1990-1993 
"lets see to log-off, BYE then return and 
I'm out". 

Search completed, no problems, some 
hesitation in perfoxnning limiting 
function. 



1/25/93 

1:40 



1:43 



1:44 



1:45 



JODI 

logged-on, some question in choosing 

" AgcQss Internet " from the main menu. 

"What were those directions again? 

Oh yes, " Access Internet ". 

"Uhh. . . Obesity and Youth should do it". 

Misspelled search items but system allowed 

backspacing to make corrections. 

"To make time-span limits. . .let's see the 

screen says to type L then 1 then 1990- 

1S22. 

Search completed, "how to get out of this 
thing", I guess, BYE". 
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1/25/93 

2;30 



2:35 



1/26/93 

1:04 



1:09 



SAMGEETA 

Logged on , hesitation due to lack of 
"computer ease" , hadn't used a computer 
very much. "Oh drat, how to get ocraputer 
to do directions, oh yes, she (researcher) 
said to use Return as Function Key". 
"BYE to get out of it". Logged off, search 
completed. 

LEAH 

Logged-on, confused in which password to 
use, search" of Card password: . No 
other problems once in system. 
Search completed. "Oh that was easy." 



1/26/93 

3:10 

3:11 



3:15 

1/27/93 

12:05 



12:10 



SLLIOfTT 

Logged-on, mistyped •Psearch", so had to 
escape to quit Firs1:Search, then re-enter, 
"Is ERIC in humanities or Social Sciences, 
let's try Social Sciences^ yes it's 
there . " 

Search completed. 
NATALIE 

Logged-on , took time to re-read search 

sheet, direction sheets on screen before 

giving any search commands. 

logged-of f , search completed and 

successful. 



1/27/93 

3:10 
3:11 



3:15 



STELLA 

Logged-on • 

"How did it say to limit searches. Oh yes 



thOT 1990-1993 . 



confusion with limxt selection, all 
else o.k. 

Logged-of f , search completed 
successful . 



some 



and 



ERLC 
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